Repository navigation
feat(ui): compare runs across workspaces on two axes, by case, and score by score - #42
Merged
Merged
Conversation
`defineScore(name, { rank })` replaces `primary`: rank 1 is the headline score, 2 the next core score, and so on;
scores without a rank are supporting scores. Two scores of one task cannot share a rank, and a rank is a whole number
from 1. The `score.defined` event adds `rank` and keeps `primary` (written for rank 1), so reports written before ranks
still open and readers keep their declaration order.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ore by score - Comparisons list and home: experiments run in several workspaces as leaderboards; only comparisons of two or more things are listed. - Compare: Y / X / baseline pickers with an axis preview beside the menu (a drawer on phones); a two-axis distribution with 95% intervals, the Pareto front, a crosshair, linear or log scales, and a click-pinned column card; differences from the baseline paired by case (an estimation plot); every score and metric as compact heat cells shared with the matrix (`HeatCell`), with the best per axis outlined and a beyond-noise tally; columns can be hidden from the charts; wide screens lay the charts out in two or three columns. - Server: comparison rows split by the matrix axes every column has (an axis only one column runs is averaged), and cells carry counted scores, numeric metrics, and the case duration. - Viewer orders scores by rank; losses take the theme hue turned half a circle (`--series-1-opposite`); heat fills stay within 50% so the page ink keeps its contrast. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Approval required for deploying to Cloudflare Workers (Preview) for docs.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Comparing the same experiment across several workspaces (for example 7 ASR rephrase products) had no entry point, labelled every column the same, merged the matrix rows of a case, and drew only a bar chart without uncertainty. This PR adds a cross-workspace comparison and charts that tell a real gap from noise.
vieval
defineScore(name, { rank })replacesprimary: rank 1 is the headline, 2 the next core score, and so on.score.definedaddsrankand keepsprimary(rank 1), so earlier reports still open. Docs updated.Server (
/api/compare/*)variant, such ascontext=none); an axis only one column runs (apromptaxis in one workspace) is averaged into apooledcell, so columns stay paired case by case.countedscores (a failed case counts 0), numericmetrics, anddurationMs; columns carryworkspaceIdandproject; the response carriesmetricDefinitions.Viewer
HeatCell), best per axis outlined, a beyond-noise tally per column;--series-1-opposite); heat fills stay within 50% (measured ≥ 5.7:1 contrast), also in the compact matrix.Testing
pnpm lint,pnpm typecheck,pnpm exec vitest run: 86 files, 712 tests passed.rank(DSL and event),orderScores,comparison.ts(intervals, paired deltas, Pareto front, labels).Follow-ups
🤖 Generated with Claude Code